Add statistical type aggregate functions, including autocorrelation, skewness, and linear regression by CoollZzz · Pull Request #17292 · apache/iotdb

CoollZzz · 2026-03-12T13:19:53Z

This pull request adds integration tests for new statistical aggregation functions in IoTDB, specifically focusing on correlation, covariance, and related statistical measures. It introduces comprehensive test coverage for these functions, updates utility constants, and expands the list of recognized aggregation functions.

New aggregation function support and test coverage:

Added new aggregation functions to BuiltinAggregationFunctionEnum: CORR, COVAR_POP, COVAR_SAMP, REGR_SLOPE, REGR_INTERCEPT, SKEWNESS, and KURTOSIS.
Added corresponding string helper methods for these functions in TestConstant (e.g., corr, covarPop, covarSamp, etc.), enabling easier test query construction.

Integration tests for statistical functions:

Added IoTDBCorrelationIT integration test class, which tests the corr (correlation) function across various scenarios: type checking, different data types, alignment by device, use in HAVING clauses, multi-device queries, group by level, and sliding window queries.
Added IoTDBCovarianceIT integration test class, which tests covar_pop and covar_samp (covariance) functions, including type and arity validation, cross-type queries, alignment by device, HAVING clause usage, group by level, and sliding window queries.This pull request adds support for several advanced statistical aggregation functions to the IoTDB query engine, including correlation, covariance, regression, skewness, and kurtosis. It introduces new accumulator classes to implement these functions and updates the relevant factory and enum classes to register and handle them appropriately.

New statistical aggregation functions:

Added new aggregation function types: CORR, COVAR_POP, COVAR_SAMP, REGR_SLOPE, REGR_INTERCEPT, SKEWNESS, and KURTOSIS to BuiltinAggregationFunctionEnum for recognition in the system.
Updated AccumulatorFactory to recognize these new functions as multi-input or single-input aggregations and to instantiate the appropriate new accumulator classes when requested. [1] [2] [3]

Implementation of new accumulator classes:

Introduced CorrelationAccumulator for correlation and covariance calculations, supporting both population and sample variants.
Introduced RegressionAccumulator for regression slope and intercept calculations.
Introduced CentralMomentAccumulator for skewness and kurtosis calculations.

Integration with sliding window aggregators:

Updated SlidingWindowAggregatorFactory to support the new statistical functions within sliding window queries.This pull request introduces support for several advanced statistical aggregation functions in the IoTDB query engine, including correlation, covariance, regression, skewness, and kurtosis. It adds new accumulator implementations for these functions and integrates them into the aggregation and sliding window frameworks.

The most important changes are:

New statistical accumulator implementations:

Added CentralMomentAccumulator for computing skewness and kurtosis. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CentralMomentAccumulator.java)
Added CorrelationAccumulator for correlation and covariance calculations. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CorrelationAccumulator.java)
Added RegressionAccumulator for regression slope and intercept calculations. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/RegressionAccumulator.java)

Integration with aggregation framework:

Updated AccumulatorFactory to support the new aggregation functions, including logic for multi-input and single-input accumulators, and to instantiate the new accumulator classes as appropriate. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/AccumulatorFactory.java) [1] [2] [3]

Sliding window support:

Modified SlidingWindowAggregatorFactory to support the new statistical functions within sliding window aggregations. (iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/slidingwindow/SlidingWindowAggregatorFactory.java)

Copilot

Pull request overview

This PR introduces several advanced statistical aggregation functions (correlation, covariance, linear regression, skewness, kurtosis) into IoTDB’s aggregation framework across both TREE and relational/table engines, and adds integration tests to validate behavior and semantics end-to-end.

Changes:

Register new aggregation types (CORR, COVAR_POP, COVAR_SAMP, REGR_SLOPE, REGR_INTERCEPT, SKEWNESS, KURTOSIS) across thrift, SQL constants, function enums, type inference, and factories.
Implement new accumulator/state logic for correlation/covariance/regression/central moments (including grouped + table accumulators and sliding-window support).
Add integration tests (TREE and relational) covering correct results, edge cases, HAVING, align-by-device/group-by-level, and sliding windows.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
iotdb-protocol/thrift-commons/src/main/thrift/common.thrift	Adds new `TAggregationType` enum values for the statistical aggregations.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/relational/TableBuiltinAggregationFunction.java	Registers new table builtin aggregation names and intermediate-type mapping.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java	Adds result-type inference and datatype validation for the new functions (including 2-input validation helper).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/SchemaUtils.java	Wires new aggregation names/types, partial suffix handling, and scan-order consistency.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/constant/SqlConstant.java	Adds SQL function-name constants for the new aggregations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/udf/BuiltinAggregationFunction.java	Registers new functions and updates multi-phase/statistics capability flags.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/metadata/TableMetadataImpl.java	Adds relational semantic checks (arity/type) and return types for the new functions.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/parameter/AggregationDescriptor.java	Adds partial-aggregation name mapping for new `TAggregationType`s.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/parser/ASTVisitor.java	Updates aggregation arity validation for new functions.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/ExpressionTypeAnalyzer.java	Adds 2-input type validation hook for multi-input statistical aggregations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableRegressionAccumulator.java	New table accumulator for regression slope/intercept.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableCovarianceAccumulator.java	New table accumulator for covariance (pop/samp).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableCorrelationAccumulator.java	New table accumulator for correlation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/TableCentralMomentAccumulator.java	New table accumulator for skewness/kurtosis.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedRegressionAccumulator.java	New grouped table accumulator for regression slope/intercept.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedCovarianceAccumulator.java	New grouped table accumulator for covariance (pop/samp).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedCorrelationAccumulator.java	New grouped table accumulator for correlation.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/grouped/GroupedCentralMomentAccumulator.java	New grouped table accumulator for skewness/kurtosis.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/AccumulatorFactory.java	Registers new table/grouped accumulators and marks new multi-input aggregations.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/slidingwindow/SlidingWindowAggregatorFactory.java	Enables sliding-window aggregation support for new statistical functions.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/RegressionAccumulator.java	New TREE-model accumulator for regression slope/intercept (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CovarianceAccumulator.java	New TREE-model accumulator for covariance (pop/samp) (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CorrelationAccumulator.java	New TREE-model accumulator for correlation (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/CentralMomentAccumulator.java	New TREE-model accumulator for skewness/kurtosis (with merge/remove).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/aggregation/AccumulatorFactory.java	Registers new TREE-model accumulators and multi-input aggregation creation.
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/IoTDBTableAggregationIT.java	Adds relational integration tests and test data for the new statistical functions.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBSkewnessKurtosisIT.java	Adds TREE-model IT coverage for skewness/kurtosis across scenarios.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBRegressionIT.java	Adds TREE-model IT coverage for regression slope/intercept across scenarios.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBCovarianceIT.java	Adds TREE-model IT coverage for covariance pop/samp across scenarios.
integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBCorrelationIT.java	Adds TREE-model IT coverage for correlation across scenarios.
integration-test/src/main/java/org/apache/iotdb/itbase/constant/TestConstant.java	Adds helper string builders for new function names in tests.
integration-test/src/main/java/org/apache/iotdb/itbase/constant/BuiltinAggregationFunctionEnum.java	Extends test-side builtin aggregation enum with new functions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ent checks

…ity by removing unnecessary comments and whitespace

…variance calculations

…ovariance, and Regression accumulators to handle state removal

…ments functions

…lators

…rrelation accumulators

Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 3 comments.

Also fix central moment merge logic to keep total counts in long and avoid precision loss when combining intermediate states.

JackieTien97

Thanks for the substantial work — the coverage is thorough (tree + table + grouped accumulators, sliding-window / window-frame support, type validation, and a good spread of ITs including multi-region merge).

I have one correctness issue I'd like resolved before merge, one semantics question, and a few minor cleanups (all left as inline comments):

Correctness

long overflow in the KURTOSIS denominator (CentralMomentAccumulator, TableCentralMomentAccumulator, GroupedCentralMomentAccumulator): (count-1)*(count-2)*(count-3) is computed in long and overflows for count above ~2.1M (sign-flips near ~2.5M), producing wrong kurtosis. Million-row groups are common here. Fix by computing in double.

Semantics

SKEWNESS (population / biased) vs KURTOSIS (sample / bias-corrected) use different estimator conventions. Please confirm intent, make them consistent, and document the exact definitions.

Minor

Tree CentralMomentAccumulator lacks the EPSILON zero-clamp that the table variants apply after element removal — can yield spurious non-null skew/kurtosis in sliding windows over constant data.
CorrelationType is a single-value enum with an unreachable guard branch (dead code; repeated in the table/grouped variants).
TypeInferenceUtils: Javadoc /** */ used as an inline comment inside a switch.
ExpressionTypeAnalyzer: a stray blank line is the only change in the file.

A couple of non-blocking notes not tied to a specific line:

No user-facing documentation update for the 7 new SQL functions — is a docs PR planned?
getDoubleValue(...) is duplicated nearly verbatim across ~9 new classes; it could be a shared helper.

Overall direction looks good; mainly the kurtosis overflow needs addressing.

Use sample-corrected skewness to align with kurtosis, avoid long overflow in kurtosis formulas, normalize tree-model central moments after removal, and clean up correlation/type-inference related reviewer comments.

…skewness, and linear regression (apache#17292)

CoollZzz changed the title ~~Enhance aggregate functions with correlation, regression, and validation~~ Add statistical type aggregate functions, including autocorrelation, skewness, and linear regression Mar 12, 2026

CoollZzz closed this Mar 13, 2026

CoollZzz reopened this Mar 13, 2026

CoollZzz force-pushed the feat/Statistical-Aggregate-Functions branch 2 times, most recently from 443e5a1 to c2f1afe Compare March 23, 2026 09:10

CoollZzz closed this Apr 1, 2026

CoollZzz reopened this Apr 1, 2026

CoollZzz force-pushed the feat/Statistical-Aggregate-Functions branch from a7d919f to 39d950d Compare April 1, 2026 10:33

JackieTien97 requested a review from Copilot April 13, 2026 01:31

Copilot started reviewing on behalf of JackieTien97 April 13, 2026 01:32 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

Comment thread .../iotdb/calc/execution/operator/source/relational/aggregation/TableRegressionAccumulator.java

JackieTien97 requested changes Apr 13, 2026

View reviewed changes

CoollZzz added 13 commits April 28, 2026 18:30

Add correlation and covariance aggregate functions to the query engine

9f5188a

Enhance aggregate functions to support TIMESTAMP data type

33a229f

Add regression aggregate functions: REGR_SLOPE and REGR_INTERCEPT

b8cc6cb

Add skewness and kurtosis aggregate functions with input validation

b216676

Refactor input validation for aggregate functions to streamline argum…

891ae63

…ent checks

Refactor aggregation function type checks to improve input validation

45c24e8

Refactor correlation and regression accumulators to improve code clar…

d79fe2b

…ity by removing unnecessary comments and whitespace

Refactor skewness,corr calculation in central moment accumulators

56d02db

Refactor correlation accumulators to use CovarianceAccumulator for co…

8f0f815

…variance calculations

Implement removeIntermediate method for CentralMoment, Correlation, C…

861124f

…ovariance, and Regression accumulators to handle state removal

Add integration tests for correlation, covariance, regression, and mo…

9b4c4cf

…ments functions

fix: address review comments for statistical aggregation functions

9255685

refactor: update package structure for statistical aggregation accumu…

0d473a9

…lators

CoollZzz force-pushed the feat/Statistical-Aggregate-Functions branch from bc31e44 to 0d473a9 Compare April 29, 2026 08:18

feat: implement removeInput method for regression, covariance, and co…

23893c5

…rrelation accumulators

JackieTien97 requested a review from Copilot May 25, 2026 01:10

Copilot started reviewing on behalf of JackieTien97 May 25, 2026 01:10 View session

JackieTien97 requested changes May 25, 2026

View reviewed changes

Comment thread integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBCorrelationIT.java Outdated

Comment thread integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBCorrelationIT.java Outdated

Copilot AI reviewed May 25, 2026

View reviewed changes

Merge tree-model statistical aggregation ITs into one class.

4080987

Also fix central moment merge logic to keep total counts in long and avoid precision loss when combining intermediate states.

CoollZzz requested a review from JackieTien97 May 25, 2026 05:49

JackieTien97 requested changes Jun 2, 2026

View reviewed changes

CoollZzz force-pushed the feat/Statistical-Aggregate-Functions branch from 13b4ca4 to 4080987 Compare June 3, 2026 08:22

CoollZzz requested a review from JackieTien97 June 3, 2026 09:27

Fix statistical aggregation semantics and reviewer issues

6409654

Use sample-corrected skewness to align with kurtosis, avoid long overflow in kurtosis formulas, normalize tree-model central moments after removal, and clean up correlation/type-inference related reviewer comments.

CoollZzz force-pushed the feat/Statistical-Aggregate-Functions branch from 9ba8157 to 6409654 Compare June 3, 2026 10:32

JackieTien97 merged commit 99bf6d4 into apache:master Jun 4, 2026
46 of 50 checks passed

MileaRobertStefan pushed a commit to MileaRobertStefan/iotdb that referenced this pull request Jun 26, 2026

Add statistical type aggregate functions, including autocorrelation, …

01827ab

…skewness, and linear regression (apache#17292)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add statistical type aggregate functions, including autocorrelation, skewness, and linear regression#17292

Add statistical type aggregate functions, including autocorrelation, skewness, and linear regression#17292
JackieTien97 merged 16 commits into
apache:masterfrom
CoollZzz:feat/Statistical-Aggregate-Functions

CoollZzz commented Mar 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JackieTien97 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

CoollZzz commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CoollZzz commented Mar 12, 2026 •

edited

Loading